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Summary: Multimedia instructional materials require learners to select, organize, and integrate information across multiple 
modalities. To facilitate these comprehension processes, a variety of multimedia design principles have been proposed. This study 
further explores the redundancy principle by manipulating the degree of partial redundancy between written and narrated content. 
Ninety high school students learned about cohesion via animated lesson videos from the Writing Pal intelligent tutoring system. 
Videos were crafted such that narrated and onscreen written content overlapped by 10%, 26%, or 50%. Across conditions, stu- 
dents gained significantly in their knowledge of cohesion-building strategies and the effects of cohesion on writing quality. How- 
ever, degree of redundancy did not influence learning gains. Additionally, although more-skilled readers outperformed less-skilled 
readers, reading skill did not interact with the degree of redundancy. These results provide evidence that a broad range of par- 
tially redundant multimedia materials may be viable instructional tools that benefit diverse learners. Copyright © 2015 John Wiley 
& Sons, Ltd. 


Instructional materials are often presented in multimedia 
formats where learners must simultaneously attend to multiple 
modalities, integrating written (i.e., visual) and spoken 
(i.e., auditory) information (Adesope & Nesbit, 2012; 
Mayer, 2009). Online courses might offer lecture materials in 
which presentations are accompanied by prerecorded narration 
(e.g., Cramer, Collins, Snider, & Fawcett, 2007). Similarly, 
many intelligent tutoring systems (ITSs) and other computer- 
based learning environments incorporate instructional text, 
diagrams, and dialogs that are spoken or explained by 
animated pedagogical agents (Craig, Driscoll, & Gholson, 
2004; Craig, Gholson, & Driscoll, 2002; Johnson, Rickel, & 
Lester, 2000). For example, ITSs such as AutoTutor (Graesser 
et al., 2004), iSTART (Jackson & McNamara, 2013), and 
Writing Pal (W-Pal; Roscoe & McNamara, 2013) use agents 
to provide spoken explanations of subject matter content, 
strategies, and feedback. Likewise, teachable agents systems, 
such as Betty’s Brain (e.g., Segedy, Kinnebrew, & Biswas, 
2013), and game-based systems, such as BiLAT (e.g., Lane, 
Hays, Core, & Auerbach, 2013) and Crystal Island (e.g., Rowe, 
Shores, Mott, & Lester, 2011), combine text and narration to 
engage learners in productive dialogs and narratives with the 
animated characters while also providing instructional 
content and feedback. 

Given the ubiquity of multimedia instructional tools that 
combine text and narration, an important area of research 
has been to establish key design guidelines for creating such 
materials. With this work, Mayer and colleagues (Mayer, 
2005; Mayer, 2009; Mayer & Moreno, 2003; Moreno & 
Mayer, 1999; also refer to Chandler & Sweller, 1991; 
Sweller, Ayres, & Kalyuga, 2011) have outlined cognitive 
theories of multimedia learning along with several principles 
for managing the learning challenges that arise. In brief, it 
has been argued that processing of visual (i.e., printed 
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text and images) and auditory (i.e., sounds and narration) 
information is handled via separate but related channels that 
are limited in capacity. Both channels rely on attention pro- 
cesses to select where to look or listen, organizational pro- 
cesses to construct coherent representations from presented 
words and images, and integration processes that merge 
concepts and representations within and across each modal- 
ity to form an understanding of the content. These active 
learning processes are essential to successful learning. Stu- 
dents must process the material deeply through inference 
generation and the organization of ideas if the objective is 
for them to remember, understand, or transfer the informa- 
tion (Eason, Goldberg, Young, Geist, & Cutting, 2012; 
Magliano & Millis, 2003; McNamara & Kintsch, 1996; 
McNamara, O’Reilly, Best, Ozuru, 2006). An important 
caveat, however, is that learners can be overwhelmed by 
multiple or competing cognitive demands. Findings from 
multimedia learning research demonstrates that the extent 
to which instructional materials require higher levels of 
selection, organization, and integration across one or more 
modalities can lead to increasing difficulty or cognitive load 
that hinders learning (Leahy, Chandler, & Sweller, 2003; 
Leahy & Sweller, 2011). Similarly, the benefits of multime- 
dia instructional materials may also depend on learners’ 
individual characteristics such as reading skill and prior 
knowledge (O’Reilly & McNamara, 2007; Ozuru, Dempsey, 
& McNamara, 2009). Consequently, optimizing learning via 
materials that require integrative processing presents a 
challenging balancing act for educators. 

A central aim for multimedia design is to support effective 
processing and comprehension by mitigating key factors that 
generate unnecessary processing demands (e.g., Mayer & 
Moreno, 2003). Specifically, educators strive to design materials 
in which comprehension processes are directed toward key 
content rather than extraneous details or interfaces. For exam- 
ple, presentations can be segmented such that information is 
presented in brief chunks with learner-controlled pacing (seg- 
mentation principle). When learners are required to engage in 
deep comprehension processes, it can be beneficial to focus 
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their time and attention on individual ideas while they gradu- 
ally develop their understanding. Similarly, instructional de- 
signers can eliminate superfluous material that only serves 
to distract the learners (coherence principle), and can use cues 
(e.g., agent gestures) to guide attention to key concepts (sig- 
naling principle). Visual and verbal materials should also be 
aligned in terms of their placement onscreen and temporal 
presentation. Essential text, images, and narrations should ap- 
pear near each other in the presentation (spatial contiguity 
principle), and corresponding text and narrations should oc- 
cur simultaneously rather than sequentially (temporal conti- 
guity principle). 

In this study, we focus upon the redundancy principle 
(e.g., Mayer & Johnson, 2008) for the design of instructional 
materials in a writing strategies tutoring system. Verbally 
redundant presentations can occur when a narrator verbalizes 
a caption that appears beneath a diagram or when an animated 
agent ‘reads aloud’ examples or text that are simultaneously 
presented onscreen. Prior research has observed mixed or 
negative findings when fully redundant (i.e., identical) 
information is presented simultaneously via text and narration 
(e.g., Adesope & Nesbit, 2012; Jamet & Le Bohec, 2007; 
Kalyuga, Chandler, & Sweller, 1999; Moreno & Mayer, 
2002). On one hand, if the visual and auditory channels are 
separate, then identical stimuli in each channel might not inter- 
fere with each other. One might even expect a positive effect if 
processing in one channel supported processing in the other. 
For instance, learners with reading difficulties may stumble 
over unfamiliar words seen in the text, but hearing those words 
spoken aloud may offer additional cues for decoding and 
comprehension (Scheiter, Schuler, Gerjets, Huk, & Hesse, 
2014). By contrast, redundant materials also place higher 
demands on two processing channels, but each stream of 
information is the same. Nothing new is gained by attending 
to and trying to reconcile both channels, and doing so merely 
places additional, unnecessary load on the learner. These 
problems may be exacerbated by presentations that also violate 
other principles, such as spatial contiguity and coherence 
(Mayer & Johnson, 2008). 

In a recent meta-analysis, Adesope and Nesbit (2012) 
considered the mixed findings regarding multimedia learning 
with redundant text and narration. They observed that 
verbally redundant materials were not more effective than 
text-only presentations (weighted mean effect size, g+ = — .04) 
although they were they were somewhat more effective than 
narration-only presentations (g+ = .29). Thus, when students 
were given textual information and explanations, the 
addition of redundant narration was of little use. However, 
students may have found narration-only presentations 
challenging because of inherent limitations of auditory chan- 
nel processing. Importantly, the majority of studies reviewed 
by Adesope and Nesbit (2012) examined highly redundant 
text-narration presentations in which information presented 
in the text and narration were identical or verbatim. Few 
studies considered partially redundant presentations with 
a lower degree of correspondence between text and narra- 
tion although these studies showed strong effects overall 
(*+ = • 99). 

Mayer and Johnson (2008) argued that fully redundant 
presentations violate the coherence principle, which states 


that extraneous details should be omitted from multimedia 
materials. In this case, offering text that merely reproduces 
the narration is unnecessary. Instead, redundant text may 
be used more sparingly to only highlight key terms, phrases, 
and explanations that are especially important (e.g., a signal- 
ing function). In two experiments, Mayer and Johnson 
(2008) observed that partially redundant text used in this 
manner resulted in significantly stronger learning outcomes 
than nonredundant presentations. More recently, Yue and 
colleagues (Yue, Bjork, & Bjork, 2013) contrasted three 
levels of verbal redundancy: no redundancy (narration only), 
partial redundancy (text summaries of the narration; partial 
overlap), and full redundancy (identical text and narration; 
-100% overlap). Their results demonstrated that both no 
redundancy and full redundancy resulted in poorer learning 
than partial redundancy. That is, when the text was only 
partially overlapping with the narration, learning outcomes 
were stronger than when the text and narration were either 
mismatched or perfectly overlapping. Ari and colleagues 
(2014) similarly found that partially redundant materials 
could be beneficial in self-paced learning of complex content. 

An important aim of the current study is to further clarify 
guidance for the design of multimedia instructional 
materials. In this study, we experimentally manipulate the 
degree of redundancy for presentations with lower 
text-narration correspondence. Specifically, instead of broad 
contrasts of no redundancy versus partial redundancy or 
perfect redundancy, this study examines three different 
levels of partial redundancy. Although evidence suggests 
that partial redundancy is ideal, we wish to better understand 
the effective range of this phenomenon. Should partially 
redundant presentations have a small degree of text-narration 
overlap (e.g., 10% overlap) or should the overlap be greater 
(e.g., 50%)? How worried should instructional designers be 
about the amount of overlap? Yue and colleagues 
(Yue et al., 2013) argued that partial redundancy could create 
‘desirable difficulty’ such that mild discrepancies between 
text and narration stimulate or facilitate active comprehen- 
sion processes of generation and integration across 
modalities. Similarly, partial redundancy may help to signal 
key ideas and direct attention (Mayer & Johnson, 2008). A 
crucial question is whether the degree of overlap influences 
such desirable difficulty or signaling and at what point (i.e., 
too much or too little overlap) the benefits may not be 
observed. 

An additional factor to consider in relation to the 
degree of redundancy may be learners’ prior reading ability 
(e.g., McNamara & Shapiro, 2005; Scheiter et al., 2014), 
which includes reading comprehension and vocabulary skills. 
When nonredundant information is presented via text and 
narration, learners must engage in comprehension processes 
to understand and integrate the two channels. Under normal 
circumstances, characteristics of the readers (e.g., reading 
ability) and text (e.g., cohesion) can interact to determine 
how well readers integrate information and build a coherent 
representation (McNamara, Kintsch, Songer, & Kintsch, 
1996; McNamara & Magliano, 2009; O’Reilly & McNamara, 
2007; Ozuru et al., 2009). Skilled readers, for example, build 
deeper understanding of texts by more regularly engaging 
in active processing (e.g., generating inferences) than less 
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skilled readers (Oakhill, Cain, & Bryant, 2003; Rapp, van den 
Broek, McMaster, Kendeou, & Espin, 2007). Presentations 
that include text and narration yield benefits when learners 
engage in active processing and integrate the two modalities 
together, suggesting more-skilled readers may be better 
equipped to comprehend multimedia presentations. This 
may be true particularly when visual and auditory channels 
are related but not redundant. 

Less-skilled readers, however, may benefit from relatively 
more redundancy between information presented through each 
modality. Adesope and Nesbit (2012) observed benefits of 
verbally redundant presentations (compared to narration-only) 
for students fluent in the language of instruction (g+ = .24), 
early or nonfluent (£+ = 0.82), or second-language learners 
(g+ = .58). Although only a very small number of studies 
examined the latter two cases, effect sizes suggested that less 
fluent readers benefit more from full redundancy. More 
directly, Scheiter et al. (2014) tested the potential moderating 
effects of reading comprehension skill when learning from 
presentations that were text-only or text with animations, and 
between presentations that were written-only, narration-only, 
or (verbally redundant) written-with-narration. Their results 
demonstrated a positive effect of the multimedia instructional 
materials (text and animation) for immediate recall, and they 
also observed that verbally redundant materials hindered learn- 
ing compared with narration-only presentations. Importantly, 
although students of higher reading ability outperformed stu- 
dents of lower reading ability, reading skills did not interact 
with multimedia or modality variables. Thus, in a direct test, 
prior reading ability did not appear to moderate the effects of 
verbal redundancy. 

One limitation of prior work is that reading ability and 
verbal redundancy were only considered in cases of high 
text-narration overlap. The relative effects of partially 
redundant materials, and how such presentations benefit 
more and less-skilled readers, remain an open question. 
Intuitively, more-skilled readers may be better equipped to 
take advantage of the signaling functions of partially 
redundant materials and to engage in deep integration 
processes to reconcile text and narration channels. Thus, 
these learners might be equally successful with minimally 
redundant (e.g., 10% overlap) to moderately redundant 
(e.g., 50% overlap) presentations. More-skilled readers might 
even benefit most from relatively lower correspondence 
because such conditions require the most active selecting and 
integrating processes. In contrast, less-skilled readers, who 
struggle with selection and integration processes in even single 
modality materials (e.g., text, McNamara & Magliano, 2009; 
McNamara & Shapiro, 2005), may find it more challenging 
to understand presentations with very low overlap (e.g., 10% 
overlap). Although these students may still benefit from 
partially redundant materials that stimulate more active 
processing, a higher correspondence (e.g., 50% overlap) 
between the text and narration may be more suitable. 

WRITING PAL 

The current study occurs within the context of W-Pal, an ITS 
developed to support adolescent students’ acquisition of 


writing strategies across three phases of the writing process 
(Roscoe, Brandon, Snow, & McNamara, 2013; Roscoe & 
McNamara, 2013; Roscoe, Varner, Weston, Crossley, & 
McNamara, 2014). The system comprises eight instructional 
modules that offer explicit strategies for prewriting ( Free- 
writing and Planning), drafting ( Introduction Building, Body 
Building, and Conclusion Building), and revising 
( Paraphrasing , Cohesion Building, and Revising). Students 
are introduced to writing strategies and mnemonic devices 
via short, animated videos narrated by pedagogical agents. 
W-Pal allows students to practice these strategies in two 
complementary ways. First, students can practice targeted 
strategies by playing a suite of educational games. Each 
module is associated with one or more games that allow 
students to practice identifying examples of strategy use or 
applying the strategies to author new text. Second, students 
can practice by writing prompt-based, argument essays 
similar to the SAT exam. Students submit their essay to the 
W-Pal system, which provides summative feedback (i.e., a 
holistic rating) and formative feedback (i.e., actionable 
recommendations for strategies and ways to revise the text). 
Such scores and feedback are driven by natural language 
processing algorithms that assess numerous linguistic 
features and properties of the text (McNamara, Crossley, & 
Roscoe, 2013). Prior research has found that interacting with 
the complete W-Pal system over a period of 2 to 3 weeks 
supports improved writing proficiency, writing strategy 
acquisition, substantive revising, and self-efficacy (e.g., Crossley, 
Varner, Roscoe, & McNamara, 2013; Roscoe, Brandon, 
et al., 2013; Roscoe, Snow, & McNamara, 2013). 

With regards to multimedia learning, a key issue pertains 
to the design of the instructional lesson videos and accompa- 
nying agent narration. W-Pal videos are generally well 
aligned with Mayer’s (2005, 2009) design principles. For 
instance, the lessons are segmented and self-paced, and 
related visual and verbal materials are both spatially and 
temporally contiguous. Prior feasibility testing quickly 
revealed the importance of removing extraneous details — 
student users responded negatively to humorous asides and 
off-topic banter between animated agents and demanded 
more concise, content-focused narration. However, although 
W-Pal lesson videos were designed to provide partially re- 
dundant text and narration, the ‘ideal’ degree of overlap 
remained unclear. To provide writing strategy instruction 
that is effective and accessible to a broad range of adoles- 
cents (e.g., students of different reading ability levels), it 
was important to understand whether the lessons may need 
to be redesigned to offer more or less redundancy. 

This context for investigating partial redundancy diverges 
in meaningful ways from commonplace features of prior 
work. First, previous research has primarily focused on 
well-defined domains such as lightning and mechanical 
systems (e.g., Mayer & Johnson, 2008) and astronomy 
(e.g., Yue et al., 2013), and the instructional materials 
typically require about 5 minutes to view the complete 
presentation. Second, previous work has overwhelmingly 
analyzed college undergraduate participants (Adesope & 
Nesbit, 2012) learning in controlled (e.g., laboratory) 
environments. In contrast, W-Pal targets high school adoles- 
cents and addresses the ill-defined domain of writing. The 
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characteristics of effective writing are often difficult to 
assess and individual authors may employ diverse strategies. 
Thus, when learning about writing and writing strategies, 
students must often grapple with problems (e.g., lack of 
cohesion) that involve multiple or subjective solutions. In 
addition, although individual W-Pal lesson videos are about 
5 minutes in length, each strategy module (e.g.. Cohesion 
Building) comprises four to five videos. Consequently, 
instruction on a given topic in W-Pal requires about 20 to 
25 minutes of viewing. Overall, the current research context 
represents an interesting departure from previous work 
that explores partial redundancy by targeting a younger 
population, a novel learning domain, and longer instruc- 
tional materials. 

Research questions 

Prior research has observed that partial redundancy between 
text and narration in multimedia presentations may be more 
beneficial than highly redundant (i.e., the redundancy effect) 
or single media presentations (i.e., the multimedia effect). 
Within the context of partially redundant presentations, this 
study explores how the degree of text-narration overlap, 
along with prior reading ability, may impact learning. Four 
questions thus guide our research and analyses: 

• Do adolescent students improve their knowledge of writing 
by studying animated lessons (i.e., partially redundant 
multimedia presentations) provided by W-Pal? 

• Are knowledge gains influenced by the degree of 
redundancy? 

• Are knowledge gains influenced by prior reading ability? 

• How do degrees of redundancy and prior reading ability 
interact to influence learning? 

METHOD 

Participants 

Participants included 90 high school students (grades 9 
through 12; primarily the 10th grade) from an urban area in 
the southwestern USA. The students were recruited via their 
English classroom teachers and were offered extra credit for 
their time. Participants were asked to provide their gender, 
age, ethnicity, year in school, and self-reported GPA. There 
were no significant differences in the distribution of 
demographic factors across conditions (i.e., p- values for all 
F or X 2 tests were >.30). Demographic data are summarized 
in Table 1. 

Learning domain 

Students learned about strategies for making their writing 
more cohesive via lessons from the W-Pal Cohesion 
Building module. This module includes four strategy lesson 
videos. In the Cohesion Building Overview lesson, an 
animated agent (Mr. Evans, a classroom teacher character) 
provides an overview of cohesion, offering a simplified 
definition of cohesion and its role in writing quality. For 
example, Mr. Evans discusses how cohesion involves 


Table 1. Gender, ethnicity, year in school, and GPA of participants 
by redundancy condition 



Partial redundancy condition 

Measure 

10% overlap 
(n = 32) 

26% overlap 
(71 = 28) 

50% overlap 
(ti = 29) 

Gender 

Female 

14 

16 

16 

Male 

18 

13 

13 

Age* 

15.3 (0.8) 

15.3 (1.0) 

15.2 (0.7) 

Ethnicity 

African-American 

5 

5 

2 

Asian 

5 

2 

1 

Caucasian 

18 

13 

17 

Hispanic 

3 

7 

5 

Year in school 

Freshman 

5 

6 

3 

Sophomore 

22 

17 

22 

Junior 

4 

5 

3 

Senior 

1 

1 

1 

GPA range 

1. 6-2.0 

2 

0 

2 

2. 1-2.5 

1 

2 

3 

2.6-3. 0 

7 

5 

3 

3. 1-3.5 

6 

12 

9 

3.6 and above 

16 

10 

12 

Reading score* 

-.06 (1.88) 

-.50 (1.61) 

.57 (1.89) 


Note. “Values reported for age and reading score are means. All other values 
are frequency counts. 


building connections between ideas such that ideas ‘flow’ 
from sentence to sentence and from paragraph to paragraph. 
Students are taught that cohesion generally results in a more 
coherent, unified, and understandable essay. Mr. Evans then 
introduces three cohesion building strategies that are further 
explained in subsequent lessons. The strategy-specific 
lessons are narrated by one of two high school student 
characters, Sheila or Mike. In the Signpost Strategy lesson, 
students are taught to identify and address undefined 
referents (e.g., words such as this and some that are not 
specified by a noun phrase). In the Connectives Strategy 
lesson, students learn how to use transition words and 
phrases (e.g., moreover and on the other hand) to show 
conceptual connections between sentences and ideas. 
Finally, students are shown how to selectively repeat key 
themes and words through a text in the Threading Strategy 
lesson. Each lesson video is at most 5 minutes in duration, 
totaling about 20 minutes of instruction. 

Conditions 

Participants were randomly assigned to one of three condi- 
tions that varied by percentage of text-narration redundancy. 
Across all conditions, the lesson video background, 
animated agents, narration scripts, and duration were held 
constant. All text appeared on a whiteboard that was depicted 
next to the agent, and the location and size of the whiteboard 
remained constant. Information content was equivalent 
across conditions, but the wording of the onscreen text varied 
experimentally (Figure 1). Partial redundancy was manipu- 
lated not by changing the quantity of text included onscreen 
but by manipulating the degree of overlap. Specifically, the 
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50% Overlap 


26% Overlap 



10% Overlap 

Figure 1. Lesson presentations maintained content equivalence and similar visual organization across 50%, 26%, and 10% overlap conditions 


50% overlap condition (n = 29) was presented with a white- 
board with text that was verbally delivered by the character. 
Not everything the character said was on the whiteboard, 
but every word on the whiteboard was said by the character 
(approximately one out of every two words). The 26% over- 
lap condition ( n = 29) was presented with a whiteboard that 
summarized lesson strategies and highlighted key terms, with 
about one of every four words overlapping. The 10% overlap 
condition (n- 32) viewed a whiteboard on which most 
content words were different from the narration. The text 
explained the same information using different words and 
sentence structure than the narration, resulting in about 1 
out of every 10 words overlapping between the script and 
onscreen text. Importantly, the overall amount of text onscreen 
remained similar across conditions (refer to Figure 1 for an 
example). 

Although these conditions represent more nuanced variations 
in redundancy than are typically examined (e.g., perfectly 
overlapping multimedia materials versus non-overlapping 
or single modality materials), the differences across condi- 
tions were still substantive. In the 50% overlap condition, 
one out of every two words overlapped between the narration 
script and the onscreen text. In the 10% overlap condition, 
such overlap only occurred for 1 out of every 10 words. 
Moreover, these variations also span commonplace and 
authentic design of multimedia materials. In practice, repro- 
duction of an agent’s entire script onscreen (100% overlap) 
would be cumbersome and inauthentic. Self-pacing and 
learner control are hindered when agents read text aloud to 
learners, and the coherence principle is violated when the vi- 
sual interface is cluttered by unnecessary words and images. 
Similarly, a lack of any supporting text or a complete 


mismatch between text and narration (0% overlap) would also 
be inauthentic compared with multimedia learning materials 
used in practice (Mayer, 2005; Mayer, 2009). In light of these 
concerns, along with the logistics of conducting classroom- 
based research with high school students, we did not include 
0% overlap and 100% overlap conditions in this study. 

Overlap percentages 

To establish the degree of overlap between the whiteboard 
text and character narration, we tabulated the number of 
words that appeared on the whiteboard and matched the 
narration. If the text and narration used different forms of 
the same word (e.g., ‘easier to understand’ versus ‘more 
understandable’), the match was counted as one-half. Like- 
wise, a match was counted as one-half if the word appeared 
in both the text and narration, but within a different segment 
of the sentence or different syntax. For example, in one case, 
the agent stated, ‘These words are like signposts that point at 
important ideas’, while the whiteboard displayed, ‘link all 
“ signpost’ ’ words to a meaningful noun or noun phrase’. 
The overlapping terms here were ‘words’ and ‘signpost’. 
However, because they are rearranged in the sentence, each 
was counted as a half-match. Non-content words, such as 
a, an, the, and, but, were included in the calculation. To 
obtain the overall degree of overlap, we divided the total 
count of matching terms by the total number of words in 
the narration script. Importantly, the amount of onscreen text 
was largely similar across conditions, and there were consis- 
tently fewer words on screen than in the narration. That is, in 
contrast to prior studies where ‘high overlap’ included longer 
verbal labels whereas ‘low overlap’ relied upon short verbal 
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labels, our conditions maintained a similar amount of text 
and visual organization. 

Time on task 

Participants in all conditions completed the same procedures, 
and the experimental manipulations did not affect the length 
of the lesson videos. Nonetheless, as with many naturalistic 
and authentic learning tasks, procedures were self-paced 
and participants could potentially adjust the amount of time 
spent studying the lessons based on the amount of redundancy 
(e.g., Ari et al., 2014). For example, learners might respond to 
low overlap by taking more time to consider the dual streams 
of information. Self-pacing effects were not observed in 
this study. Across conditions, participants appeared to spend 
an average of just over an hour completing their assigned tasks 
(i.e., 74 minutes in the 50% overlap condition, 69 minutes in 
the 26% overlap condition, and 75 minutes in the 10% 
overlap condition). These minor differences in time-on-task 
were not statistically significant, F< 1.00. 

Measures 

Gates-MacGinitie reading test 

Reading comprehension skill was tested with the Gates- 
MacGinitie (4th ed.) reading skill test (form S) level 10/12 
(MacGinitie & MacGinitie, 1989). The test consisted of 48 
multiple-choice questions assessing students’ comprehension 
of 1 1 short passages. Each passage was associated with two 
to six questions, which measured shallow comprehension as 
well as deeper comprehension that required the reader to 
make inferences about the text. The participants were 
administered the standard instructions with 20 minutes to 
complete the test. The vocabulary section of the Gates- 
MacGinitie (4th ed.) test (form S) level 10/12 (MacGinitie 
& MacGinitie, 1989) was used to assess vocabulary skill. 
The test comprised 45 sentences or phrases, each with an 
underlined vocabulary word. For each underlined word, 
participants were asked to select the most closely related 
word from a list of five choices. The items were designed to 
suggest the word’s part of speech but provide no contextual 
information about meaning. Participants were administered 
the standard instructions and given 10 minutes to complete 
the test. An examination of raw mean vocabulary scores 
(M = 28.6, SI) = 8.4) suggested that students’ vocabulary 
skills were fairly strong (i.e., a score range of approximately 
20-25 is indicative of a grade equivalency range of 9-12). 
By contrast, raw mean comprehension scores (M = 22.9, 
577 = 9.9) suggested that students’ comprehension skills were 
at or slightly below grade level (i.e., a score range of approx- 
imately 22-30 is indicative of a grade equivalency range of 
9-12). Overall, participants demonstrated a range of reading 
skill comparable with normal high school performance, but 
with good vocabulary knowledge. 

For purposes of analysis, students’ scores on each subscale 
were transformed into standard deviation units and summed, 
thus producing a single score representing overall reading 
ability (Table 1). A one-way ANOVA comparing reading 
scores across all conditions was not statistically significant, 
F(2,87) = 2.61, p=. 079. Flowever, pairwise comparisons 
suggested that, despite random assignment, students in the 


26% overlap condition had lower prior reading ability than 
did students in the 50% overlap condition (p = .026). 

Strategy knowledge test 

Students answered three open-ended questions assessing their 
knowledge and understanding of cohesion and cohesion-building 
strategies. Question 1 asked, ‘Cohesion is an important aspect of 
writing. In your own words, please give a definition of cohesion.’ 
Question 2 probed students’ understanding of the impact of 
cohesion, ‘How does cohesion influence the quality of an essay?’ 
Finally, Question 3 directly addressed strategy knowledge, 
‘What strategies can be used to improve the cohesion of an 
essay?’ Students answered these questions both before and after 
instruction. Importantly, these questions did not simply test re- 
call or recognition of the video contents, but instead tapped spe- 
cific knowledge and comprehension about cohesion in writing. 

To score students’ responses, the researchers first 
reviewed the instructional videos (i.e., agent narrations along 
with onscreen text and illustrations) to extract key idea units 
pertaining to each question. For example, with regards to the 
definition of cohesion, the materials explained that cohesion 
is the linking or bridging of ideas, across sentences and 
paragraphs, which results in unified themes and arguments. 
Using these idea units as a template, we reviewed students’ 
responses to understand how students expressed these ideas 
in their own words. For instance, students might not use 
terms like ‘linking’ or ‘bridging,’ but instead describe 
‘joining two ideas smoothly’ or ‘how all the sentences relate 
or tie together’. Because students’ responses to any of the 
questions could potentially address definitions, impact, or 
strategies (e.g., explaining how cohesion influences essay qual- 
ity while also describing a cohesion-building strategy), all three 
responses were reviewed together for scoring purposes. 
Students could receive credit for any given idea unit only once, 
even if that idea was stated in multiple question responses. 

The coding scheme was iteratively tested and refined. Two 
of the researchers independently coded an initial set of 19 
student responses and then discussed and resolved discrep- 
ancies. This process was repeated with a different set of 16 
student responses. A final set of 16 students’ responses were 
then independently coded and assessed for inter-rater reliability. 
Overall, the coders agreed on 96.6% of the codes, yielding a 
Kappa of 0.91. We determined that this was an acceptable 
agreement and a single coder finished the remaining 
responses. The final coding scheme included template 
responses along with exemplars of acceptable alternative 
expressions (Table 2). Students could earn up to 2 points 
for defining cohesion, 4 points for describing the influence of 
cohesion on writing quality, and 4 points for describing strategies 
for building cohesion (i.e., maximum total score of 10). 

Procedure 

Participants completed the entire study, including instruction 
and assessments, via their home computers. The study was 
completed in a single session using a web-based survey 
service. Each participant completed one of three surveys. 
Each survey began with the same pretest questions. Next, 
students watched the lesson videos, which varied based on 
text-narration overlap (i.e., 10%, 26%, or 50% overlap). 
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Table 2. Template and exemplars for coding student responses to knowledge questions 


Idea units 

Description and examples of idea units 

Definition of cohesion 

Linking of ideas 

Ideas are connected across sentences and paragraphs 
‘Cohesion is the connection between sentences’ 

‘it’s how you connect ideas between sentences and paragraphs’ 

Unity 

Ideas and themes are unified across the whole text 
‘Cohesion if making parts of your writing come together as a united whole.’ 
‘It’s a form of unity within a piece of literature.’ 

Impact on quality 

Flow 

Smooth flow from one idea to the next 
‘Flowing smooth’ 

'Being able to make ideas flow’ 

Readability 

Text can be read easily by the reader 
‘A cohesive paper is easy to read’ 

‘clearly written’ 

Understandability 

Main ideas can be understood by the reader 
‘makes your essay make more sense’ 

‘It makes it easy to understand’ 

On-topic 

Ideas seem related to one another without tangents or digressions 
‘Keeps an essay strong by being on topic.’ 

‘Cohesion is the binding topic of an essay, instead of going of tangent’ 

Cohesion strategies 

Connectives 

Use of transition words and phrases to link ideas 
‘Connectors can be used to improve cohesion.’ 

‘For example, poeple should what my class calls “transition words’” 

Signpost 

Defining referents (e.g., ‘this’); avoiding vague terms and phrases 
‘Clarifying “signposts’” 

‘Look for words such as “that, this most” explain what those things are’ 

Threading 

Repeating key terms and ideas across sentences and paragraphs 
‘Referring back to or building on top of a previous sentence’ 

‘Threading strategy — “sew” the words together’ 

Other 

Other strategies that improve cohesion indirectly (e.g., outlining) 

‘Planning also helps you write a cohesive essay as well.' 

‘Reading a paper out loud for flow’ 


Finally, participants completed the posttest questions. On 
average, the study required just over 1 hour to complete. 


RESULTS 

Overall learning gains 

An initial analysis simply examined overall student learning 
across all conditions. A one-way, repeated-measures 
ANOVA was conducted to determine whether students’ 
knowledge of cohesion improved from pretest to posttest 
(i.e., an effect of instruction). As shown in Table 3, students 
increased significantly in their knowledge of cohesion from 
pretest (M= 2.07) to posttest (M= 2.90). It is important to 
note that these scores and gains are rather low relative to 


Table 3. Mean pretest and posttest knowledge assessment scores 



Pretest 

Posttest 




Measure 

Mean 

SD 

Mean 

SD 

F 

P 

d 

Total score 

2.07 

1.26 

2.90 

1.76 

28.60 <.001 

.54 

Definition of cohesion 

0.27 

0.49 

0.32 

0.54 <1.00 

.339 

.11 

Impact on quality 

1.22 

0.88 

1.53 

0.97 

7.51 

.007 

.33 

Cohesion strategies 

0.58 

0.56 

1.04 

0.95 

24.09 <.001 

.60 


the maximum possible score; cohesion is a difficult concept 
to teach and understand at the high school level. Nonethe- 
less, these gains are meaningful (d= .54) given that students 
received only 20 minutes of instruction without the opportu- 
nity to practice the strategies in their own writing. 

Subsequently, we sought to tease apart students’ under- 
standing of cohesion related to their definition of cohesion, 
knowledge of the role cohesion plays in writing quality, and 
cohesion-building strategies. We found that students generally 
struggled to define cohesion at pretest (M- 21) and posttest 
( M=.49 ) and did not gain significantly. However, students 
did appear to gain knowledge about the impact of cohesion 
on writing quality from pretest (M=122) to posttest 
(M= 1.53) and cohesion-building strategies from pretest 
( M=.58 ) to pretest (M= 1.04). W-Pal’s primary focus is to 
provide students with actionable strategies that they can imple- 
ment to improve their writing. Thus, these results suggest that 
W-Pal was somewhat successful in communicating such infor- 
mation to students. Nonetheless, future iterations of W-Pal 
might provide further elaboration to better explain the funda- 
mental nature of cohesion. 

Influence of degree of overlap and prior reading ability 

The effects of the degree of overlap and prior reading ability 
were first tested via a 2 (instruction) x 3 (degree of overlap) 


Copyright © 2015 John Wiley & Sons, Ltd. 


Appl. Cognit. Psychol. 29: 669-679 (2015) 


676 


R. D. Roscoe et al. 


mixed, repeated-measures ANCOVA. Instruction was a 
within-subjects variable (pretest vs. posttest), and degree of 
redundancy was a between-subjects variable (50% vs. 26% 
vs. 10% overlap). Prior reading ability was included as a 
covariate. To further test for a potential three-way interaction 
among the three variables, a follow-up analysis used a 
median split to divide students into ‘higher prior reading 
ability’ and ‘lower prior reading ability’ groups. Subse- 
quently, a 2 (instruction) x 3 (degree of overlap) x 2 (reading 
ability) mixed-repeated measures ANOVA was conducted. 
For clarity, Tables 4-6 display only means and standard 
deviations based on these follow-up analyses. Because of 
the observed differences across knowledge types, analyses 
of redundancy and reading ability were conducted separately 
for each knowledge type. 

Learning about the definition of cohesion 
For students’ ability to define the concept of cohesion, we 
observed an effect of prior reading ability, F(\ ,86) = 10.28, 
p = .002, but observed no main effects of instruction or 
degree of overlap (both Fs< 1.00). Thus, as noted earlier, 
students overall did not improve in their knowledge of the 
definition of cohesion. In addition, students struggled with 
this concept regardless of the degree of overlap in the multi- 
media presentation. However, students with stronger reading 
ability demonstrated a better grasp of the basic nature of 
cohesion. No significant interactions were observed in either 
the initial or follow-up analysis (Table 4). 


Learning about the impact of cohesion 
For students’ knowledge of how cohesion impacts writing 
quality, significant main effects were observed for instruc- 
tion, HI ,86) = 7.45, p = .008, and prior reading ability, 
F( 1 ,86) = 17.04, p < .001, but no main effect was observed 
for degree of overlap (F < 1). No significant interactions 
were observed in either the initial or follow-up analyses 
(Table 5). Thus, students in this study appeared to acquire 
knowledge about how cohesion influences the quality of an 
essay, but such gains did not vary as a function of either 
partial redundancy or prior reading ability. Similar to 
findings for definitions of cohesion, students who possessed 
stronger reading skills also demonstrated better overall 
knowledge about the impact of cohesion. 

Learning about cohesion-building strategies 
For students’ knowledge of strategies for making their 
writing more cohesive, we observed significant main effects 
for instruction, F(l,86) = 24.80, p< .001, and prior reading 
ability, F(l, 86)= 19.65, p < .00 1 , but observed no main 
effect of degree of overlap (F < 1) (Table 6). Interestingly, 
a significant interaction was observed in the initial analysis 
between learning gains and prior reading ability, F(\ ,86) 
= 5.24, p = .024. These effects were further examined in the 
follow-up analysis. Although students with lower reading 
ability gained from pretest (M=. 42, SD = .50) to posttest 
(M=. 71, SI) = .82), students with higher reading ability 
appeared to learn somewhat more about cohesion strategies 


Table 4. Mean pretest and posttest scores for definition of cohesion as a function of degree of overlap and prior reading ability median split 





Pretest 



Posttest 

Degree 

Lower prior reading ability 

Higher prior reading ability 

Lower prior reading ability 

Higher prior reading ability 

overlap 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Mean SD 

50% 

.07 

.27 

.47 

.64 

.14 

.36 

.73 .80 

26% 

.12 

.34 

.31 

.48 

.13 

.35 

.31 .48 

10% 

.27 

.46 

.35 

.61 

.12 

.34 

.47 .51 

Table 5. 

Mean pretest and posttest scores for impact of cohesion 

as a function of degree of overlap and prior reading ability median split 




Pretest 



Posttest 

Degree 

Lower prior reading ability 

Higher prior reading ability 

Lower prior reading ability 

Higher prior reading ability 

overlap 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Mean SD 

50% 

.64 

.74 

1.67 

.72 

1.57 

.85 

1.73 .96 

26% 

1.25 

.86 

1.31 

.75 

1.00 

.89 

1.70 1.05 

10% 

.67 

.98 

1.70 

.68 

1.33 

1.11 

1.92 .76 

Table 6. 

Mean pretest and posttest scores for cohesion strategies 

as a function of degree of overlap and prior reading ability median split 




Pretest 



Posttest 

Degree 

Lower prior reading ability 

Higher prior reading ability 

Lower prior reading ability 

Higher prior reading ability 

overlap 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Mean SD 

50% 

.57 

.51 

1.00 

.53 

.78 

.80 

1.47 .83 

26% 

.31 

.48 

.69 

.48 

.62 

.72 

1.00 .91 

10% 

.40 

.51 

.53 

.62 

.73 

.96 

1.59 1.06 
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from pretest (M=. 73, SD=. 58) to posttest (M- 1.38, 
SD = .96). The interaction was marginally significant, 
F(l,86) = 3.60, p = . 061. No other two-way or three-way 
interactions were found. 


DISCUSSION 

Multimedia instructional materials must be carefully designed 
to engage students in active comprehension processes that are 
focused upon important content (Mayer, 2005, 2009). Within 
this framework, high levels of verbal redundancy — the extent 
to which visual materials (e.g., text and diagrams) and auditory 
materials (e.g., narration and sound effects) are closely 
overlapping in content — have been shown to hinder learning 
(e.g., Jamet & Le Bohec, 2007; Leahy et al., 2003; Moreno 
& Mayer, 2002). When the exact same information is 
presented in both visual and auditory channels, learners may 
be forced to waste cognitive resources attempting to process 
two identical streams for little benefit. By contrast, partially 
redundant presentations are those in which verbal redundancy 
is used purposefully to signal key terms and concepts. That is, 
core ideas and explanations presented in the narration might be 
reinforced using redundant text on screen (e.g., a summary), 
whereas less important points or details might be omitted from 
the text. Studies of partial redundancy have found that such 
presentations lead to stronger learning gains than either nonre- 
dundant or fully redundant presentations (Mayer & Johnson, 
2008; Yue et al., 2013). Partially redundant materials may 
facilitate learners’ active selection, organization, and integra- 
tion of ideas by marking where and when students need to 
pay attention while providing just enough discrepancy to 
stimulate integration across modalities. 

In this study, we examined the effects of varying degrees 
of partial redundancy. Although prior research has 
established that partial redundancy can be beneficial, it was 
unclear whether variations in the amount of partial redun- 
dancy would impact learning. For example, how ‘minimal’ 
might the text-narration overlap be and still benefit learners? 
In addition, we considered whether an ‘optimal’ degree of 
partial redundancy might differ on the basis of students’ 
prior reading ability. Various levels of redundancy may not 
be suitable to all learners. For instance, although more- 
skilled readers might benefit from low overlap that fosters 
integration across modalities, less-skilled readers might only 
experience these benefits with a bit more signaling (i.e., more 
redundancy). Past research has suggested that students of all 
reading abilities levels can learn from multimedia presenta- 
tions (Adesope & Nesbit, 2012) but reading ability may not 
moderate the effects of redundancy (Scheiter et al., 2014). 

Our results suggest that variations in the degree of partial 
redundancy do not significantly influence learning gains. 
Across all three levels of partial verbal redundancy (i.e., 10%, 
26%, and 50% overlap), students improved significantly in their 
ability to describe the impact of cohesion on text quality and 
specific strategies for building cohesion. One possible explana- 
tion is that the manipulation was too fine-grained. Perhaps the 
variations in verbal redundancy were too subtle to have a 
meaningful impact. In practice, however, differences across 
our experimental conditions were not subtle (Figure 1). In one 


condition, the overlap between text and narration was 1 out of 
every 10 words (10%), whereas other conditions demonstrated 
overlap of one out of every four words (26%) or one out of 
every two words (50%). These are substantive variations. In 
some ways, this is a promising finding because it means that a 
broad range of partially redundant presentations may be useful 
for multimedia instruction. ITS developers and instructional 
designers may not need to worry about finding a ‘perfect’ level 
of overlap but may instead focus on providing the right content, 
the best scaffolds, and other design principles (e.g., coherence 
and relevance). Nonetheless, a potential contrast for future 
research may be to examine a larger disparity between partially 
redundant materials (e.g., 80% overlap versus 20% overlap). 
Additional fine-grained manipulations may also be useful in 
determining optimal presentation format, such as explicitly 
varying both sentence fonnat (key words or complete sentences) 
and type of redundancy (exactly matching words or using 
synonyms). 

Also in accord with prior research, we observed a strong 
effect of prior reading ability, such that more-skilled readers 
possessed significantly more knowledge about cohesion than 
less-skilled readers. Reading skill is a complex phenomenon 
that comprises word-level knowledge and skills (e.g., recog- 
nizing words, decoding unfamiliar, and breadth of vocabulary) 
along with text-level knowledge and skills (e.g., understanding 
text structure and being able to bridge ideas across text 
sections) (e.g., Oakhill & Cain, 2012). The current study 
further highlights the differences between more-skilled and 
less-skilled readers with respect to their understanding of text 
flow and cohesion. At both pretest and posttest, more-skilled 
readers demonstrated more knowledge about cohesion. Thus, 
understanding of text cohesion may constitute one aspect of 
skilled readers’ text expertise. Finally, in the case of strategy 
knowledge — the focus of W-Pal content and instruction — 
stronger reading ability also seemed to foster learning from 
the multimedia presentations. It is likely that students who al- 
ready possessed some foundational knowledge about cohesion 
and reading were better prepared to extend such strategy 
knowledge to the domain of writing. 

Current results also provide additional support for past 
findings that reading ability may not moderate the effects 
of verbal redundancy. There was no interaction between 
reading skill and learning with partially redundant presenta- 
tions. Although more-skilled readers outperformed their 
less-skilled counterparts, both groups of students gained 
overall. This result suggests that strategy instruction using 
animated agents and multimedia presentations can benefit 
students of diverse reading levels. One possibility, as 
suggested earlier, is that the manipulation of redundancy 
was too subtle. Students’ reading abilities may be more 
sensitive to a starker contrast between high and low text- 
narration overlap. However, within the range of partially 
redundant materials used in this study, reading ability did 
not seem to moderate the effects of text-narration cohesion. 
This finding again suggests that ITS developers and 
instructional designers have some flexibility when creating 
multimedia presentations. One important goal for ITS 
developers is to identify student characteristics that can 
usefully guide the delivery of instructional content. The 
current findings suggest that reading ability may have 
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limited use in personalizing these kinds of presentation 
formats for different students. 

One note of caution of these results pertains to the range of 
students recruited for the study. The majority of participants 
were ‘A’ or ‘B’ average students (i.e., GPA greater than 3.0 
on a 4.0 scale) and thus may have benefitted from stronger 
general knowledge or other academic skills. For example, 
although our less-skilled and more-skilled readers differed 
from each other significantly, both groups may have repre- 
sented generally capable readers. Inspection of the raw 
Gates-MacGinitie scores, however, suggested that students 
were well within the normal range of high school students 
for reading comprehension but had good vocabulary knowl- 
edge. In continued research, it may important to specifically 
sample from more diverse populations with a wider range of 
ability, including struggling readers, second-language 
learners, or readers with learning disabilities. Just as future 
research may need to examine a broader disparity between 
degrees of text-narration redundancy, this work may also 
need to consider a broader range of academic skills or other 
potential learning challenges. 

Another contribution of this work relates to the W-Pal 
tutoring system, which specifically emphasizes writing 
strategy instruction and knowledge. These data provide 
additional evidence that W-Pal can support writing strategy 
knowledge acquisition (Roscoe, Brandon, et al., 2013). 
Although students received only a limited portion of what 
W-Pal offers — students in this study viewed only the lessons 
without opportunities to play strategy practice games or 
author practice essays — they were able to gain knowledge 
about cohesion. These gains might have been stronger in 
the context of additional practice and feedback with the 
complete system. Additionally, W-Pal currently provides 
only a brief introduction on the nature of cohesion, which 
students struggled to articulate both before and after instruc- 
tion. Thus, one improvement to the system may be to offer 
more explicit instruction about the underlying principles of 
cohesion. 

A final contribution speaks to the instructional domain and 
target population. Whereas the majority of past work on 
verbal redundancy and multimedia presentations has been 
at the post-secondary level (Adesope & Nesbit, 2012), the 
current study targeted high school aged adolescents. Our 
findings suggest that partially redundant multimedia materials 
are beneficial for younger learners. Similarly, although a 
handful of multimedia studies have targeted reading (e.g., Diao 
& Sweller, 2007), the large majority have addressed science and 
technical domains such as computer literacy (Craig et al., 2004), 
anatomy (Ari et al., 2014), astronomy (Yue et al., 2013), and 
meteorology (e.g., Moreno & Mayer, 2002). In contrast to 
well-defined learning domains, wherein key concepts and 
procedures are generally constrained or well-known, writing is 
an ill-defined learning domain. Writing assessment is often 
subjective and challenging, and diverse authors may employ 
distinct yet effective strategies, hideed, recent research has 
observed that skilled writers may adopt very different 
approaches for composing high-quality essays (Crossley, 
Roscoe, & McNamara, 2014). Thus, the current work represents 
a valuable extension of research of multimedia learning into yet 
another important learning domain. The benefits of multimedia 


instructional materials do not only apply to college students 
studying well-defined domains. 
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